The Use of Complexity Metrics in Testing CA Gen Projects

Projects that do not have sufficient testing resources to perform 100% testing of new and changed programs may wish to focus their testing efforts on programs that are more likely to contain errors. Simple statement count metrics are compared with Cyclomatic Complexity counts and these are correlated with historical data from a large Gen application to understand the relationship between these metrics and the number of changes that have affected each program. The results indicate that Cyclomatic Complexity is a useful indicator of the probability that a program will contain errors and hence a useful metric to use in prioritising testing effort. Introduction The objective of this study is to investigate the usefulness of complexity metrics in managing the testing of CA Gen developed applications. IET have developed a code coverage testing tool (pathvIEw), which measures how complete the testing is for each program (action block) based on the percentage of statements that have been executed during testing. Ideally the testing of the application would ensure 100% test coverage of all changed programs, i.e. every statement in the code is thoroughly tested. However this can often not be achieved, especially without using comprehensive automated testing software and associated test scripts, and when testing resources are limited. For situations where testers wish to focus their testing efforts, this study aims to establish whether there are any metrics that would indicate which programs are more likely to contain errors This study will also be comparing results of various methods of calculating complexity and the differences between them and the benefits of each one, as well as less established methods that have been suggested as more useful. Complexity Metrics A complexity metric produces a measure of how complex a program is, based on factors such as lines of code and types of statement. The simplest metric is Lines of Code (LOC) which is a simple count of the number of lines in the source program. Whilst often used because it is so simple to calculate, and perhaps modified to only count logical lines of code (i.e. ignoring comments), there are many disadvantages of LOC, for example, it is dependent on coding style and each type of statement has equal weighting. For CA Gen action blocks, number of statements is the nearest equivalent to LOC and is perhaps more accurate because of the highly controlled structure of the action block syntax. Cyclomatic Complexity is a measure of how complex a program is, and was originally developed by Thomas McCabe in 1976. The Cyclomatic Complexity is a count of the number of linearly independent paths within a program. For instance a simple linear program that has no decision points has a complexity of 1, whereas if it contained an IF statement, then there are two separate paths through the code and so it would have a complexity of 2. A way of calculating the Cyclomatic Complexity of a program is to look at its control flow graph. The formula for the complexity is: M= (E-N) +2P where: M= complexity, E= the number of edges in the graph, N= the number of nodes, P= the number of connected components (always 1 for a single program). A easy formula of calculating Cyclomatic Complexity for CA Gen action blocks is Number of Decisions +1. A decision in this context is a possible branch in the code, and the method adopted in this study is to increment the count by one for each of the following types of statements: IF ELSE IF REPEAT – UNTIL WHILE FOR CASE Database exception clause (except for when successful) The original McCabe method treated each branch as adding one to the complexity count. A variation of this method (extended cyclomatic complexity) includes Boolean operators in the count. In this study we refer to the original method as CC1 and the extended method as CC2. For example, the statement: IF a=1 AND b=1 adds a complexity of +1 in CC1, but +2 in CC2. The study compared the results of an additional two types of method, CC3 which only increments the count by one for each CASE OF clause and ignores each individual CASE clause and CC4 which counts distinct IF/ELSEIF statements, i.e. if the same IF statement is present multiple times, it is only counted once. An advantage of CC4 is that logic that repeats the same test multiple times (for The complexity of this program is 3. E=9 N=8 P=1 M= (9-8) + 2, M=3 Edges example debug statements nested within IF debug = ‘Y’ do not have an artificially high complexity count. The table below summarises each of the four methods used in the study. Statement Type CC1 CC2 CC3 CC4 IF / ELSE IF +1 +1 for IF/ELSEIF and +1 for each AND/OR clause +1 +1 for each distinct IF/ELSEIF clause REPEAT – UNTIL +1 +1 +1 +1